Effectiveness of Automated Function Testing with Petri Nets: A Series of Controlled Experiments

Existing work has developed techniques for automated generation of function tests from high-level Petri nets. Yet there is no empirical evidence that demonstrates the costeffectiveness of this approach. This paper presents a series of controlled experiments to evaluate the fault detection capabilities of various strategies for test generation from high-level Petri nets. We built test models and generated executable test code for three subject programs. Then we executed the test code against more than 300 mutants of the subject programs. Each mutant is a variation of the original subject program with one injected fault. The experiment results show that (a) the reachability coverage-based tests are more effective than the tests for state coverage and for transition coverage, (b) postcondition-based test oracles are more effective than state-based oracles, (c) robustness tests with invalid inputs are critical to improving fault detection capability. In particular, the reachability coverage-based tests together with robustness tests killed 99.7% of the mutants. Keywords— Software testing, model-based testing, Petri nets, fault injection, test coverage, test oracle